Fusion Models for Improved Image Captioning

Visual captioning aims to generate textual descriptions given images or videos. Traditionally, image models are trained on human annotated datasets such as Flickr30k and MS-COCO, which limited in size diversity. This limitation hinders the generalization capabilities of these while also rendering them liable making mistakes. Language can, however, be vast amounts freely available unlabelled data have recently emerged successful language encoders coherent text generators. Meanwhile, several unimodal multimodal fusion techniques been proven work well for natural generation automatic speech recognition. Building recent developments, with aim improving quality generated captions, contribution our this paper is two-fold: First, we propose a generic model framework caption emendation where utilize different strategies integrate pretrained Auxiliary Model (AuxLM) within traditional encoder-decoder visual frameworks. Next, employ same Masked (MLM), namely BERT, model, viz. Show, Attend, Tell, emending both syntactic semantic errors captions. Our experiments three benchmark datasets, Flickr8k, Flickr30k, MSCOCO, show improvements over baseline, indicating usefulness proposed strategies. Further, perform preliminary qualitative analysis emended captions identify error categories based type corrections.